Modern computing systems are capable of running a large variety of software applications and platforms. A computing system can provide an operating system allowing applications executing on the computing system to access and interact with hardware components of the computing system. Through this interaction, the software applications can accept user input and provide output, which can be presented to the user through a graphical user interface presented on a display device. Additionally, the output can be forwarded to a remote computing device.
Remote computing technologies allow users to use a computing system without being physically located at the computing system. Through remote access, users can connect to a remote computing system over a public or private network. The user can use a local computing system to send input to the remote computing system and the remote computing system can respond. Although not physically present at the remote computing system, the user can interact with the remote computing system as though the user was using the system directly.
Additionally, virtualization technologies have become widely used for sharing limited computer hardware resources. In a hardware virtualization environment, sometimes also referred to as a platform virtualization environment, one or more virtual machines can simulate a host computer with its own operating system. In particular, software executed on these virtual machines can be separated from the underlying hardware resources. As a result, virtual machines running on a same host computer can each have its own operating system. For example, a computer that is running Microsoft Windows operating system may host a virtual machine that simulates a computer running a Linux operating system, based on which Linux-based software can be executed.
Further, virtualization technologies can provide remote access to computer resources across public or private networks. Users can use a variety of clients to access a virtual desktop or applications running on a host computer in the virtualized environment. These clients can be directly connected to the host computer or they can connect to the virtualized environment through a public or private network. The client can capture user input and transfer it across the network to the host computer's virtual environment. The application receiving the user input can respond by performing some action and presenting a response to the user. Virtualization allows multiple users to share hardware resources instead of requiring that each user connect to a dedicated remote computing system.
Applications running in a remote, local, or virtualized computing environment can respond to user input updating the output for display on a display device. An application can provide the operating system with instructions to update the graphical user interface. The operating system can then process the commands and display the update on a graphical interface. In many current and past system architectures, applications could write directly to a frame buffer through the operating system. In addition to the frame to be displayed, the operating system could provide the raw draw commands and additional information about the updated display frame such as the content type of the displayed content, the modified region of the output display frame, and other metadata about the output display frame. This information can be useful for remote technologies because instead of simply sending a bandwidth intensive graphic representing a visual display over a network, the system could combine the raw draw commands and the additional information about the display frame to optimize data sent to the remote computing environment. This optimization allowed for a much better user experience while limiting the network bandwidth required for remote computing.
Modern operating systems have moved away from past display mechanisms, relying instead on offloading rendering of the user interface to graphics processing units. As a result, the operating system receives updates from the applications, composites those updates into a display frame using the graphics processing unit, and provides a graphic or bitmap representing the composited frame to the display device or graphics system. The graphics system of the computing environment only receives a final composited display frame without any additional metadata or draw commands. Accordingly, a graphics driver forwarding the display to a remotely connected user can no longer benefit from the previously provided information, and there exists a need for improving the efficiency of the processing, encoding and delivery of display frames in virtual and remote computing environments.